A Deep Non-Negative Matrix Factorization Neural Network

Recently, deep neural network algorithms have emerged as one of the most successful machine learning strategies, obtaining state of the art results for speech recognition, computer vision, and classification of large data sets. Their success is due to advancement in computing power, availability of massive amounts of data and the development of new computational techniques. Some of the drawbacks to these deep neural networks are that they often require massive amounts of observed data, their feature representations are hard to interpret and they are not well mathematically understood when they will work, and why. Other strategies for data representation and feature extraction, such as topic modeling based strategies, have also recently progressed. Topic models, such as NMF, combine data modeling with optimization to learn interpretable and consistent feature structures in data. Previously criticized for their computational complexity, it is now possible to quickly perform topic modeling on massive streaming data sets. We introduce a deep non-negative matrix factorization framework capable of producing interpretable hierarchical classification of many types of data. Our proposed framework shows that it is possible to combine the interpretability and predictability of topic modeling learned representations with some of the power and accuracy of deep neural networks. Furthermore, we uncover a new connection between sparse matrix representations and deep learning models by combining multiple layers of NMF with a non-linear activation function and pooling, optimized by backpropagation.